Evaluation of the National Population Health Study of Neurological Conditions 2009-2010 to 2013-2014

Prepared by
Office of Evaluation
Health Canada and the Public Health Agency of Canada

January 2015

Table of Contents

• Executive Summary
• 1.0 Evaluation Purpose
• 2.0 Program Description
  • 2.1 Program Context
  • 2.2 Program Profile
  • 2.3 Program Logic Model and Narrative
  • 2.4 Program Alignment and Resources
• 3.0 Evaluation Description
  • 3.1 Evaluation Scope, Approach and Design
  • 3.2 Limitations and Mitigation Strategies
• 4.0 Findings
  • 4.1 Relevance: Issue #1 - Continued Need for the Program
  • 4.2 Relevance: Issue #2 - Alignment with Government Priorities
  • 4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities
  • 4.4 Performance: Issue #4 - Achievement of Expected Outcomes (Effectiveness)
  • 4.5 Performance: Issue #5 - Demonstration of Economy and Efficiency
• 5.0 Conclusions
  • 5.1 Relevance Conclusions
  • 5.2 Performance Conclusions
• 6.0 Lessons Learned
• Appendix 1 - NPHSNC Component Studies
• Appendix 2 - Description of the Logic Model for the National Population Health Study on Neurological Conditions
• Appendix 3 - Summary of Findings
• Appendix 4 - Evaluation Description
• Appendix 5 - Interim Performance Study
• Appendix 6 - References

List of Tables

• Table 1: Program Resources
• Table 2: Limitations and Mitigation Strategies
• Table 3: Variance Between Planned vs Actual Spending

Executive Summary

This evaluation covered the National Population Health Study of Neurological Conditions (NPHSNC) for the period from 2009-2010 to 2013-2014. The evaluation was undertaken to fulfill the requirements of the Financial Administration Act and the Treasury Board of Canada's Policy on Evaluation (2009).

Evaluation Purpose and Scope

The purpose of the evaluation was to assess the relevance and performance of the NPHSNC program. This was a scheduled evaluation for 2014-2015 on the Health Canada and the Public Health Agency of Canada Five-Year Evaluation Plan.

Program Description

The NPHSNC was a four-year study to fill knowledge gaps with respect to the incidence, prevalence, and impact of neurological conditions in Canada. The NPHSNC focused on 14 neurological conditions, with some coverage of four additional conditions. Eighteen projects were funded through the NPHSNC, including three national surveys developed by the Public Health Agency of Canada and conducted by Statistics Canada, 13 research projects, an expansion of the Canadian Chronic Disease Surveillance System, and a microsimulation platform to project the economic and human impacts of neurological conditions. The study was a one time $15 million investment intended to end in March 2013; however, it was extended by one year for reporting purposes. The initiative sunsetted in March 2014.

CONCLUSIONS - RELEVANCE

Continued Need

In 2006, the WHO estimated that neurological conditions were affecting up to one billion people worldwide and were the cause of death of an estimated 6.8 million people each year. These numbers were projected to increase as the population aged. In Canada, little research was available at that time about the incidence, prevalence and impact of neurological conditions. The Public Health Agency's NPHSNC addressed a knowledge gap in understanding the prevalence, incidence and impact of neurological conditions on Canadians, their families and caregivers. And while the study addressed pre-identified gaps, it also identified new knowledge gaps (e.g., the lack of data on specific populations such as First Nations, Inuit and Métis communities, children, and those with less prevalent neurological conditions).

Alignment with Government Priorities

The NPHSNC was aligned with federal priorities and is consistent with several recently announced initiatives in the federal priority area of neurodegenerative conditions. Generating knowledge and understanding the burden and impact of neurological conditions was, and continues to be, a priority for the Government of Canada and the Public Health Agency of Canada.

Alignment with Federal Roles and Responsibilities

The NPHSNC is aligned with the federal government role in surveillance and research as well as roles in the provision of national leadership and support for knowledge translation. It is also consistent with the role of the Public Health Agency of Canada in the prevention of chronic diseases. Federal leadership of the NPHSNC enabled the identification of gaps in knowledge and surveillance activities and the lack of standardized national estimates/projections; it also increased the visibility of the impact of neurological conditions.

CONCLUSIONS - PERFORMANCE

Achievement of Expected Outcomes (Effectiveness)

Findings indicate that the NPHSNC achieved its intended immediate outcome of stakeholder engagement. The study has also demonstrated the benefits of a collaborative governance model that may have applicability to other initiatives within the Agency. The unique governance model that featured co-leadership of the study by the Public Health Agency of Canada and National Health Charities Canada (NHCC) was widely praised. The NPHSNC brought together a broad cross-section of stakeholders who were engaged at various junctures through the planning, implementation and reporting phases of the study. This approach enhanced existing partnerships or led to the development of new ones between key stakeholders, resulting in increased knowledge development and exchange. Many informants noted the importance of bringing together a number of diverse researchers and neurological charities as key to the success of the study.

In terms of meeting the intermediate outcome related to the use of NPHSNC knowledge products, the NPHSNC has developed planned knowledge products and there are early indications that progress is being made towards achievement of this outcome. An important legacy of the study is enhancements to PHAC's surveillance for four neurological conditions: Alzheimer's disease and other dementias, Epilepsy, Parkinsonism and Multiple sclerosis. Further, the key study product - the Synthesis Report Mapping Connections: An Understanding of Neurological Conditions in Canada - was released with some delays, and is now being broadly disseminated. Efforts are now underway by study partners to promote the results of the study to stakeholders.

Although it is too soon to determine the longer-term impact of the NPHSNC on improved program planning, service and policy capacity, there is a potential to have an impact in this area as well.

Demonstration of Economy and Efficiency

Study expenditures were generally aligned with allocated funds. While the governance model and broad engagement of stakeholders during the implementation and synthesis phases were labour intensive with associated costs, the NPHSNC also leveraged expertise and volunteer time from experts, researchers and health charities. Grouping the 18 neurological conditions in a single study (compared to separate data collection for each individual condition) also contributed to efficiency. Where challenges were noted, these often had to do with delays (in funding, in conducting research, and in receiving results from the microsimulation project) which impacted the implementation of the component studies and a small number of studies which did not achieve the intended objectives or value for money. Performance indicators will be examined in the future, given the recent release of the Synthesis Report.

LESSONS LEARNED

Formal recommendations are not being proposed given that program funding ended in March 2014. Nevertheless, based on the findings and conclusions outlined in this evaluation report, there are lessons learned that may be valuable to consider in the future for similar program initiatives.

1. The NPHSNC governance model was a success story that could be applied to other Public Health Agency of Canada research initiatives. This approach enhanced existing partnerships or led to the development of new ones between key stakeholders, resulting in increased knowledge development and exchange.
2. The Public Health Agency of Canada developed the knowledge translation plan in partnership with NHCC and co-authored the report and a number of knowledge products. However, including knowledge translation in the original contribution agreement could have enhanced this process and improved implementation of this phase.
3. Research timelines should adequately anticipate (or include flexibility to deal with) start-up delays related to government processes and ethical approvals in order to ensure research initiatives are able to meet objectives.

1.0 Evaluation Purpose

The purpose of the evaluation was to assess the relevance and performance of the National Population Health Study of Neurological Conditions (NPHSNC) (2009-2010 to 2013-2014). This is a scheduled evaluation as per the Public Health Agency of Canada/Health Canada's approved Five-Year Evaluation Plan 2014-2015 to 2018-2019 to meet Financial Administration Act requirements and the Treasury Board of Canada's Policy on Evaluation (2009).

2.0 Program Description

2.1 Program Context

It is estimated that hundreds of thousands of Canadians are affected by neurological conditions (diseases, disorders and injuries to the brain and nervous system). As the Canadian population ages, the prevalence of many of these conditions will significantly increase. This is consistent with World Health Organization (WHO) predictions for increases in the burden of neurological disorders.

Neurological conditions represent a significant impact on the Canadian economy, health care system, and caregivers; however, the breadth of this impact is not well understood. There are gaps in knowledge about the overall incidence, prevalence, risk factors and impact of neurological conditions in Canada. These gaps are impeding the actions of governments and health charities devoted to developing prevention programs and providing services to those living with these conditions.

On June 5, 2009, the Minister of Health announced funding for a $15 million four-year study of neurological diseases in the Canadian population (2009-2013). The study was a one-time investment intended to end in March 2013; however, it was extended by one year for reporting purposes. The initiative sunsetted in March 2014.

2.2 Program Profile

Study Objectives

The NPHSNC was a four-year study to fill knowledge gaps with respect to the incidence and prevalence of neurological conditions, identify the use of health services and risk factors, and determine the impact of neurological conditions in Canada. The objectives of the NPHSNC were to: a) determine the scope (incidence, prevalence, and comorbidities) of neurological diseases in Canada; b) determine the personal, economic and health service impact of neurological diseases in Canada; and c) develop knowledge on risk factors for neurological diseases in Canada.

Strategic Objectives

The results were to provide essential information which can be used for policies, programs and services in the prevention and management of neurological diseases by federal, provincial and territorial governments, non-governmental organizations, regional health planning bodies, health care institutions, and health care providers.

Program Delivery

The NPHSNC was comprised of four components or focus areas:

1. Impact: Impact of Neurological Conditions on Individuals, Families and Communities. This component of the NPHSNC examined the impact of neurological conditions on Canadians, and concentrated on the more detailed experiences of individuals with neurological conditions, their caregivers and families.
2. Health Services: Existing Health Services, Gaps, Tools and Promising Practices. While not their primary focus, several research projects provided data on the health services needs for both those living with neurological conditions and their caregivers. This component addressed the use of health services, gaps in services and recommended improvements.
3. Scope: Incidence, Prevalence and Co-morbidities. This component had as its overall goal to determine the scope of neurological conditions in Canada. In order to accomplish this goal, studies were carried out to determine the incidence, prevalence and co-morbidities of the following neurological conditions^{Footnote i}:
   • Amyotrophic lateral sclerosis (ALS or Lou Gehrig's disease);
   • Alzheimer's disease and other dementias;
   • Cerebral palsy;
   • Dystonia;
   • Epilepsy;
   • Huntington's disease;
   • Hydrocephalus;
   • Multiple sclerosis;
   • Muscular dystrophy;
   • Parkinson's disease;
   • Spina bifida; and
   • Tourette syndrome.
4. Risk Factors: Learning More About the Risk Factors for Neurological Conditions. Risk factors for neurological conditions were explored through the NPHSNC. Risk was assessed from a broad perspective, including both the risk of developing a disease and the risk of poor health outcomes for those with a neurological condition.

In total, 18 projects were funded through the NPHSNC, including three national surveys conducted by Statistics Canada, 13 research projects, an expansion of the Canadian Chronic Disease Surveillance System to gather administrative data on neurological conditions, and a microsimulation platform to project the economic and human impacts of neurological conditions over the next five, ten, fifteen and twenty years. A list of the NPHSNC component studies is provided in Appendix 1.

A Synthesis Panel was formed in 2013 to bring all the available study findings together and to facilitate their uptake by researchers, service providers, policy makers and government. A Synthesis Report summarizing the key findings was written and a detailed dissemination plan to inform stakeholders (including the public) of the study results and recommendations is currently being implemented.

Stakeholders

The stakeholders for the NPHSNC were those who were directly involved in the development and implementation and those who will be interested in the results of the research. These stakeholders include:

• Public Health Agency of Canada;
• Health Canada;
• Canadian Institutes of Health Research (CIHR) and other research bodies;
• Neurological Health Charities Canada (NHCC) and their constituent organizations;
• Provincial and Territorial governments;
• Canadians affected by neurological conditions;
• The general public;
• Neuroscience researchers; and
• Health care institutions and health care providers.

Governance

The Public Health Agency of Canada is the lead federal department responsible for the management of program resources and the coordination of activities. Management and oversight of the study activities was the responsibility of an Implementation Committee made up of representatives from the health portfolio of the Government of Canada (Public Health Agency of Canada, Health Canada and CIHR) and from NHCC, a coalition of organizations that represent people with neurological conditions. A Scientific Advisory Committee, made up of nine academics, along with a representative from the Public Health Agency of Canada and one from the Neurological Health Charities Canada, provided the Implementation Committee with expert scientific advice as well as independent ongoing oversight and evaluation of the scientific elements of the NPHSNC.

2.3 Program Logic Model and Narrative

The connection between the activity areas, outputs and the expected outcomes for the logic model is described in Appendix 2^{Footnote ii}. The evaluation assessed the degree to which the defined outputs and outcomes have been achieved.

Immediate Outcome

Improved engagement in addressing gaps in the understanding of neurological conditions in Canada is expected to emerge from a coordinated approach by grouping neurological conditions together, enhancing partnerships and leveraging expertise.

Intermediate Outcome

Improved engagement is anticipated to result in the following intermediate outcome: use of NPHSNC knowledge products. It was expected that key stakeholders would use the resulting knowledge products to understand issues related to the scope, impact, risk factors, and gaps in health services for various neurological conditions. The Synthesis Report on neurological conditions in Canada was intended to serve as a key knowledge product aimed at addressing knowledge gaps regarding the number of people affected, the causes of these conditions and their cost and impact on families and caregivers. Information materials for individuals and families with neurological conditions was expected to be used to inform understanding of how to cope with the challenges of these conditions.

Longer-Term Outcome

Increasing understanding of the scope, impact, risk factors and use of health services for neurological conditions among stakeholders in Canada through the completion of various component-specific studies and comprehensive reporting, will contribute to the following long-term outcome: capacity to plan programs, services and policies to prevent or manage neurological conditions.

The Synthesis Report on neurological conditions was intended to provide critical knowledge for government and health stakeholders to reduce the burden of neurological conditions on Canadians. Policy development was to be informed through more accurate information on the incidence and prevalence of neurological conditions in Canada, as well as the individual and economic costs associated with diagnosis and care. The documented existing services and gaps were likely to benefit the neurological stakeholder community in the planning of services. Gaps in knowledge were impeding the action that governments and health charities could take to provide service and support to people living with neurological conditions.

2.4 Program Alignment and Resources

The program was part of the Agency's Program Alignment Architecture (PAA): Program 1.2 Health Promotion and Disease Prevention, Sub-program 1.2.3 Chronic (non-communicable) Disease and Injury Prevention.

The program's financial data for the years 2009-2010 through 2013-2014 are presented below (Table 1). Overall, the program had a budget of $15 million over five years.

3.0 Evaluation Description

3.1 Evaluation Scope, Approach and Design

The scope of the evaluation covered the period from 2009-2010 to 2013-2014, and included all components of the NPHSNC. The evaluation issues were aligned with the Treasury Board of Canada's Policy on Evaluation (2009) and considered the five core issues under the two themes of relevance and performance, as shown in Appendix 4. Corresponding to each of the core issues, specific questions were developed based on program considerations and these guided the evaluation process.

An outcome-based evaluation approach was used for the conduct of the evaluation to assess the progress made towards the achievement of the expected outcomes, whether there were any unintended consequences and what lessons were learned.

The Treasury Board Policy on Evaluation (2009) also guided the identification of the evaluation design and data collection methods so that the evaluation would meet the objectives and requirements of the Policy. A non-experimental design was used.

Data for the evaluation was collected using various methods, which were: document and file review and key informant interviews. More specific detail on the data collection and analysis methods are presented in Appendix 4. In addition, data were analyzed by triangulating information gathered from the different methods listed above. The use of multiple lines of evidence and triangulation were intended to increase the reliability and credibility of the evaluation findings and conclusions.

3.2 Limitations and Mitigation Strategies

Most evaluations face constraints that may have implications for the validity and reliability of evaluation findings and conclusions. The following table outlines the limitations encountered during the implementation of the selected methods for this evaluation. Also noted are the mitigation strategies put in place to ensure that the evaluation findings can be used with confidence to guide any future program planning and decision making.

4.0 Findings

4.1 Relevance: Issue #1 - Continued Need for the Program

The Public Health Agency's NPHSNC addressed a knowledge gap in understanding the prevalence, incidence and impact of neurological conditions on Canadians, their families and caregivers. Canadian and international studies, as well as the views of key informants, agreed that the burden of neurological conditions was significant, although these conditions were underresearched.

In 2006, the WHO estimated that neurological conditions were affecting up to one billion people worldwide and were the cause of death of an estimated 6.8 million people each year^{Endnote 1}. These numbers were projected to increase as the population ages. A 2007 Canadian Institute for Health Information (CIHI) report echoed the conclusions of the WHO report, calling for collective action on neurological conditions^{Endnote 2}. In Canada, it was estimated that up to one million people are affected by neurological conditions, and it was reported that neurological conditions were a leading cause of disability (or reduced ability to function), costing the economy an estimated $9 billion a year (Standing Committee on Health, March 1, 2012).

In spite of the concern about the impact of neurological conditions on Canadians, according to key informants, prior to the NPHSNC, estimates were only available for a select number of conditions, not standardized, and/or extrapolated from United States or European Union data.

Prior to the inception of the NPHSNC, a review of research on neurological conditions conducted by NHCC (in government as well as private and public foundations) concluded that the 18 conditions chosen were "very much underfunded and under researched"^{Endnote 3}. This finding was echoed in comments from almost all internal and external stakeholders. Specifically, there was a need for knowledge about incidence and prevalence, risk factors, as well as the cost and impact of the conditions on individuals and their caregivers. While the study filled a number of pre-identified gaps, many new knowledge gaps were identified:

• Canadians living in varying social and economic situations;
• First Nations, Inuit and Métis individuals, with a need for fulsome representation of each of these Aboriginal groups;
• Children, both those affected by a neurological condition and those living with an affected parent or sibling;
• Formal caregivers for individuals with severe, debilitating neurological conditions;
• Canadians who were excluded from surveys such as the Canadian Community Health Survey and the Survey on Living with Neurological Conditions in Canada; and
• Canadians with less prevalent neurological conditions, such as those outside the scope of this Study or those affected by less common conditions such as ALS, Huntington's disease, and dystonia^{Endnote 4}.

Key informants noted that the NPHSNC was a good first step but that there was still a need to address neurological research gaps, such as risk factors of neurological conditions, that would inform policy and program priority setting.

4.2 Relevance: Issue #2 - Alignment with Government Priorities

Generating knowledge and understanding the burden and impact of neurological conditions was, and continues to be, a priority for the Government of Canada and the Public Health Agency of Canada. The NPHSNC was thus aligned with federal priorities and was consistent with several recently announced initiatives in the area of neurodegenerative conditions.

In October 2008, Prime Minister Stephen Harper made a commitment to fund a national study of neurological conditions and on June 5, 2009 the Minister of Health confirmed the government's commitment by announcing $15 million in funding for a four-year NPHSNC. The Minister announced the NPHSNC by explaining that "This study will arm us with the knowledge we need to help alleviate some of the burden associated with these conditions (for individuals, families and society overall). It will allow both government and health charities to better develop and plan programs and health services". Between these two announcements, the Agency partnered with NHCC, the Canadian Institutes of Health Research and Health Canada to seek stakeholder input on priority areas for the study.

The government continues to show that neurological conditions are a priority. For example, in 2011 funding in the amount of $100 million over six years was provided to the Canadian Brain Research Fund, a national non-profit organization devoted to supporting brain research. More recently, the 2013 Speech from the Throne and subsequent announcements in 2014 supported federal leadership in addressing neurological conditions, particularly later onset conditions (e.g., Alzheimer's disease, Parkinson's disease):

• CIHR led the Canadian Consortium on Neurodegeneration and Aging (CCNA), a research hub for neurodegenerative disorders announced as part of the Economic Action Plan 2014. The CCNA received $31.5 million of funding over five years from the Government of Canada (through CIHR and a group of 13 partners from the public and private sectors, including the Alzheimer Society of Canada and Fonds de recherche du Québec - Santé) .
• Canada co-hosted a G7 Global Dementia Legacy Event with France, which aimed to foster new approaches to addressing the challenges of dementia (September 2014).
• The National Dementia Research and Prevention Plan was also announced in 2014 and the Health Minister further noted working with the Alzheimer Society of Canada to bring the "Dementia Friends"^{Footnote iii} program to Canada.

Within the Agency, Strategic Horizons 2013-2018 identified surveillance and dissemination of knowledge as key priorities^{Endnote 5}. This document noted the priority of the Agency to "enhance public health surveillance of non-communicable (chronic) disease risk factors, maternal and child health, injuries and infectious diseases...to support its needs and those of its partners to inform public health decision-making". As identified in the Health Promotion and Chronic Disease Prevention Branch and CCDP strategic plans and operational plans, and corroborated by key informants, there was strong alignment between priorities and current activities.

4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities

The federal government has a clear role to conduct surveillance and research, provide leadership and support knowledge translation. A key role of the Public Health Agency of Canada is to contribute to the prevention and reduction of risks for chronic diseases and injuries and (broadly) to facilitate national approaches to public health policy and planning. Activities of the NPHSNC were aligned with this role. Duplication was addressed through engagement of key health stakeholders in the planning and governance of the NPHSNC.

The Agency's role in health promotion and protection, surveillance and disease prevention is outlined in the Department of Health Act (2006) and the Public Health Agency Act (2006). Its more specific roles to conduct research, surveillance and knowledge exchange are presented under the Integrated Strategy on Healthy Living and Chronic Disease (ISHLCD) and neurological conditions policy and program authorities. The ISHLCD provides a framework for the federal government to promote the health of Canadians and reduce the impact of chronic disease in Canada^{Endnote 6}. A foundation of the ISHLCD is working with diverse partners in a focused way as well as enhanced collaboration among organizations, jurisdictions and sectors. The ISHLCD aligns federal public health action based on three interrelated pillars across the spectrum of health and chronic disease:

• Promotion of health by addressing the conditions that lead to unhealthy eating, physical inactivity and unhealthy weights;
• Prevention of chronic disease through action on major chronic diseases and their risk factors; and,
• Support for the early detection of chronic diseases.

The objectives of the NPHSNC align with the Agency's role in the prevention of chronic diseases, surveillance and the facilitation of national approaches to public health policy and planning through working with partners.

The federal government's convenor role in the implementation of the NPHSNC was unique because it helped bring together NGOs and the federal government to work towards a common purpose. Key informants indicated that without federal leadership on the NPHSNC, there would continue to be gaps in knowledge and surveillance activities, a lack of standardized national estimates/projections, and continued limited visibility of the impact of neurological conditions. Most key informants agreed that the role of the Agency in the NPHSNC was complementary to the role of others who were working in the area of neurological conditions. Other organizations were mainly conducting basic research on the physiology of neurological conditions (e.g., CIHR funded researchers and research institutes/universities). According to key informants, any potential overlap with existing surveillance and research conducted by other organizations was minimized through the Health Portfolio approach (the Public Health Agency of Canada, Health Canada and CIHR were members of the study Implementation Committee). NHCC representation on the committee also helped ensure that research was complementary with efforts of other organizations and sectors. Finally, the planning of the NPHSNC was guided by a rigorous and far-reaching stakeholder consultation process and inventory of existing literature and data sources to identify research gaps and needs where the study could make a meaningful difference.

4.4 Performance: Issue #4 - Achievement of Expected Outcomes (Effectiveness)

4.4.1 To what extent has the immediate outcome been achieved?

Immediate outcome #1: Improved neurological community engagement

The NPHSNC utilized a unique governance model that involved the Public Health Agency of Canada and NHCC as study co-leads, with support from Health Canada and CIHR on the Implementation Committee. This resulted in optimizing expertise and networks. This model has the potential to be applied to other areas of the Agency, other disease areas, and other government departments.

The management of the NPHSNC was based on a unique collaboration between the Government of Canada and the neurological conditions community. NHCC, a coalition of neurological charities, was the co-lead for the study with the Agency. Under a contribution agreement with the Agency, NHCC was provided capacity to participate in the management and coordination of the NPHSNC. The role of NHCC was to: co-chair the Implementation Committee; develop and support research projects and advisory teams; garner consumer input into other project plans; and keep the community of neurological stakeholders informed about the study activities and results. During the synthesis and reporting phase, the Stakeholder Engagement Panel and the Synthesis Panel included representation from the federal health portfolio, as well as members of NHCC and other stakeholders.

Despite some early challenges with governance^{Endnote 7}, the evidence from this evaluation and the Interim Performance Study demonstrated that the governance model for the NPHSNC was successful. The majority of internal and external key informants characterized the study process as a "great collaboration", "very successful", "well-managed", and "well-coordinated". Respondents perceived that the strengths of the Agency (information/data, surveillance and epidemiological expertise) were complemented by those of NHCC. The specific contributions of NHCC reportedly were:

• providing input on reports;
• making reports more accessible to a broader audience by disseminating study information and products through member organizations' networks;
• providing an important link to the "human or personal side" of neurological conditions, thereby helping to translate knowledge to the Canadian public; and
• providing disease expertise and experience (scientists, managers, strategists).

In addition to the co-management of the study, there was also praise for the role of the Scientific Advisory Committee which was viewed as having a positive intermediary role as an arm's length sounding board to monitor projects and engage the project leads in working through their challenges and identifying potential solutions.

Key to the success of the governance model, according to program documents and key informants, was early, frequent and open communications, clear roles, responsibilities and expectations, resolving issues as they arose, documenting decisions, and having shared interests and a common goal. A high level of commitment on the part of all stakeholders was also frequently noted as a prerequisite for success.

Many internal and external key informants believed that the governance model had the potential to be replicated in other areas of the Agency, other disease areas and other government departments. The Interim Performance Study noted the same, but recommended that some revisions to the model be undertaken (including strengthening the factors of success mentioned above) and ensuring that accountability for financial management was clear and the appropriate tool was used to fund collaborating NGOs to play their role in the study.

The stakeholder community was engaged in the NPHSNC and contributed to increased capacity, broadened networks, and availability of comprehensive information for stakeholders. This engagement was seen as a foundation to enable more ongoing collaborative efforts in this area.

Evidence from the document review was supported by key informants who noted that there was extensive engagement of the stakeholder community during all phases of the NPHSNC. NHCC played a key role in the planning, execution and reporting of study results and researchers and stakeholders were engaged through a variety of mechanisms (meetings, panels, online survey).

Study planning. To determine the scope and research questions for a possible study on neurological diseases, NHCC, in collaboration with the Agency and health portfolio partners, conducted an online consultation with researchers, caregivers and persons with neurological diseases in February 2009. In March 2009, a scientific and technical workshop was jointly held by the Agency and NHCC that engaged neuroscience researchers and neurological health practitioners. These consultations identified the priorities for research that became the foundation for the design of the NPHSNC.

Study implementation. In total, the funded projects included 127 researchers, 26 of whom were not only working on their own projects, but were also working as co-investigators or collaborators on other projects (leading the coordination and knowledge exchange between teams). Researchers represented 31 institutions and were in most provinces (British Columbia, Alberta, Manitoba, Ontario, Quebec, Newfoundland and Labrador, Nova Scotia and Prince Edward Island).

To encourage collaboration among researchers, a secure site was made available on mybrainmatters.ca. In addition, the Agency and NHCC co-hosted annual face-to-face progress meetings to allow participants to network and share knowledge about their respective studies and to provide advice to the NPHSNC leadership. The meetings included representatives from the research community as well as patient organizations.

The stakeholder engagement survey noted that most members of the Scientific Advisory Committee and researchers found the progress meetings to be useful in facilitating collaboration among researchers. Nevertheless, some stakeholders expressed concerns about the overall efficiency of the exercise (i.e. two days for the meetings may have been too long, presenting twice in two days to overlapping audiences) and also suggested that communications could be enhanced even further (e.g., through periodic newsletters, more information about how the results of the studies would be synthesized).

Reporting. The collaborative process for the development of the NPHSNC Synthesis Report was planned and documented through a Process Terms of Reference. The Terms of Reference described the key elements involved in synthesising the results of the NPHSNC component studies and demonstrated that stakeholders were consulted at multiple key junctures during the preparation of the report. The report was produced by a Synthesis Panel drawn from members of the Implementation Committee, the Scientific Advisory Committee, NHCC stakeholders and other representatives. Key study themes and draft reports were revised with input from stakeholders through a consultation process. A Stakeholder Engagement Panel was struck to review the report on the stakeholder consultations and make recommendations for integrating their insights in the report on scientific findings. The report itself is a Public Health Agency of Canada and NHCC co-authored product.

The Interim Performance Study and feedback from key informants indicated that stakeholders were pleased with their involvement in the study. An online survey of stakeholders conducted as part of the Interim Performance Study found that stakeholders generally agreed that there was sufficient common ground amongst them to work together and that there was agreement on the goals and objectives for the NPHSNC. Further, in spite of a few communication and collaboration challenges across the three groups (Implementation Committee, the Scientific Advisory Committee, and Principal Investigators/research teams), 85% strongly agreed or agreed that working collaboratively through the NPHSNC allowed them to achieve more than they would have achieved working alone.

According to external key informants, the NPHSNC offered the opportunity to bring together those with common interests in different expertise areas and to share information freely and respectfully. Relationships were built between researchers and patient advocates, among NGOs, among researchers, and between researchers and the Agency. The survey of stakeholders conducted for the Interim Performance Study found the same; virtually all Scientific Advisory Committee and Implementation Committee members and researchers agreed that the NPHSNC created opportunities for these types of collaborations. These collaborations have resulted in ongoing collaborations, ongoing discussions regarding collaborative research projects, transferring models across disease areas and across jurisdictions, ongoing consensus building on disease prevalence/incidence estimates, and seeking expert advice through their expanded networks.

Evaluation key informants also confirmed that the mechanisms for engagement of their community were appropriate and well-supported, and represented a well-rounded cross-section of the neurological conditions' stakeholder communities. Among the minority of key informants who identified potential gaps in the stakeholder representation, these absences were related to specific conditions (stroke, aphasia and neurodevelopment disorders affecting children) or an underrepresentation of policy-makers who they perceived were not sufficiently involved, yet are one of the key users of the information.

The evaluation and Interim Performance Study found that frequent opportunities to meet and work together through the NPHSNC contributed to an increase in the capacity of many stakeholder organizations (e.g., access to new knowledge and understanding of prevalence and patient and caregiver impacts, and research expertise). This was seen to be especially valuable for smaller, volunteer-run organizations that lack internal research capacity or resources. Most stakeholders surveyed as part of the Interim Performance Study (89%) agreed or strongly agreed that "working collaboratively as part of the Neurological Study is an asset for my organization".

Most stakeholders felt there was at least the potential for ongoing collaborations as a result of contacts and networks established through the NPHSNC and there are ongoing discussions to explore the need for more research or emerging ideas that were generated by the study. Examples include: collaborative research projects; transferring models/tools across disease areas (as a result of recognizing commonalities) and across jurisdictions; ongoing consensus-building on disease prevalence/incidence estimates; and reaching out to their expanded network of study contacts to seek expert advice.

4.4.2 To what extent has the intermediate outcome been achieved?

Intermediate outcome #1: Increased knowledge and uptake of neurological knowledge products by target populations

The NPHSNC has developed all planned knowledge products. Preliminary tracking of access to these products is underway. However, it was too early to determine the level of uptake with certainty, given the release of the material only occurred in September 2014.

A variety of knowledge products have been developed, both for the NPHSNC overall (synthesis products), as well as products generated by the individual studies. The key product of the study is the Synthesis Report - Mapping Connections: An Understanding of Neurological Conditions in Canada. The report was released with a Ministerial announcement on September 11, 2014. Mapping Connections is the culmination of the NPHSNC, summarizing the evidence that was gathered from all funded studies under each of the four study focus areas: impacts of neurological conditions; health services for neurological conditions; scope (prevalence and incidence) of neurological conditions; and risk factors for neurological conditions. For each of the focus areas, knowledge gaps were identified to point the way forward for future research.

A dissemination plan for Mapping Connections was developed by the Implementation Committee, and is currently being implemented. The report is available for download on the Agency website, and linked on the NHCC and some of its member websites. The link to the report has also been circulated via mail distribution lists, stakeholders' newsletters and social media (Facebook postings, tweets). Partnerships were key to increased dissemination: Public Health Agency of Canada, NHCC, principal investigators and other stakeholders were all involved in disseminating the study results.

There are early indications that the knowledge products were being accessed. In the first month after the release of Mapping Connections, the English version of the report had been downloaded 1,454 times, while the French version had been downloaded 197 times; the number of page views for the Synthesis Report homepage posted on the Agency website was 3,275. The average time users spent on the NPHSNC report homepage (3 minutes and 13 seconds) is above average for the amount of time typically spent on a PHAC web page (1 minute and 51 seconds).

The initial release of Mapping Connections has been enhanced by the development of other complementary knowledge translation and exchange products. A separate contribution agreement was established with NHCC following the end of the study to conduct knowledge translation and exchange activities and develop products tailored to fit the interests of various audiences (researchers, patient community, policy-makers). While there was a delay in funding NHCC between the conclusion of the NPHSNC and the initiation of the knowledge translation and exchange phase, the development of products is now well underway. Examples include a Mapping Connections brochure and postcard, Fact Sheets, slide decks, tool kits, YouTube videos, videographics, presentations at conferences/workshops/seminars.

In addition to Mapping Connections and its complementary products, each of the individual studies that were funded under the NPHSNC was required to have a knowledge translation and exchange plan. The types of knowledge translation and exchange products generated by these studies include peer-reviewed publications, conference and presentations/events/meetings, technical reports, mass media products (website postings, videos, fact sheets) for broader audiences and online learning modules/webinars. This activity is ongoing by the NPHSNC study teams.

The opinions of key informants were positive overall about the creation and initial uptake of knowledge about neurological conditions. They highlighted both the importance of stakeholders and researchers working together during the implementation of the study to increase awareness across the projects, and the combined efforts of federal and NHCC study members to increase dissemination (e.g., each using their networks). A few key informants noted the unique Public Health Agency of Canada and NHCC co-branding of Mapping Connections and complementary products, and mentioned that other efforts such as joint and individual presentations (e.g., to representatives within provincial Ministries, members of research/practitioner communities) were also taking place. Finally, through CIHI there will be ongoing access by graduate students and academic researchers to some of the datasets created by NPHSNC studies.

There were a few cautions around the knowledge translation and exchange phase of the study. External stakeholders expressed some concerns about the marketing efforts around the report (from a cumbersome study acronym to unfortunate timing of the release of the report on a busy day of announcements and limited marketing and follow-up).

In December 2014, the Program Performance Section in CCDP administered a web-based Stakeholder Satisfaction Survey with known neurological stakeholders (n=118, response rate was 44%). Results from the survey demonstrate high levels of awareness, satisfaction with, and use or anticipated use of, the Mapping Connections report. Key findings include: 93% of respondents who had read the report strongly agreed or agreed that they were satisfied; 95% of respondents strongly agreed or agreed that Mapping Connections contributed to the knowledge base related to neurological conditions; and 98% of respondents reported having already used the report or intended to use the report (e.g., citing or planning to cite the report in their work; planning to discuss and/or sharing the report with others; and using or anticipating to use the report to inform programs, policies and practices). Respondents main purposes for using the report were: efforts to improve health and service needs of people with neurological conditions and their families; to leverage funds or partner resources; to conduct additional research; to influence policy; and for education and awareness.

Overall, these early satisfaction and use findings demonstrate very high levels of satisfaction with the report, which is often found to increase the likelihood of use and application of evidence in practice, policy and program decision-making.^{Endnote 8,Endnote 9} The findings related to use are particularly notable given that the report has only been available for 3 months.

4.4.3 To what extent has the longer-term outcome been achieved?

Longer-term outcome #1: Improved capacity to plan programs, services and policies to manage and prevent neurological conditions

While it was too early to measure the impact of the NPHSNC on improved program planning, service and policy capacity, there is a potential to have an impact in this area.

The NPHSNC study made two key contributions to capacity building in the area of knowledge about neurological conditions:

• the availability of reliable and/or standardized prevalence, incidence and impact estimates and projections of neurological conditions; and
• increasing capacity for the ongoing surveillance and study of neurological conditions with the addition of four major neurological conditions to the Public Health Agency of Canada's Canadian Chronic Disease Surveillance System and guidelines for neurological disease/patient registries. This increase in surveillance capacity was an important component of the sustainability of the NPHSNC and will ensure that reliable data will be available to understand trends in the incidence and prevalence of Alzheimer's disease/other dementias, Parkinsonism, Epilepsy and Multiple sclerosis^{Footnote iv}.

According to many key informants, the public release of the NPHSNC results was too recent to demonstrate achievement of meaningful impacts on programs, services and policies. It was unrealistic to see an impact within the timeframe of the NPHSNC. As well, a few respondents noted that the Agency does not hold the levers to effect positive change in many of these health delivery areas, which is the purview of the provinces and territories (which have been informed of the results of the study). However, a number of key informants believed the study has the potential to contribute to informing decision-making in: health investments based on estimated and projected public health impacts, setting direction for policy, and informing planning for care programs and services to better meet patient needs. Critical for external stakeholders has been the availability of evidence-based estimates which are now available to health charities to raise awareness about neurological conditions and inform their advocacy strategy (e.g., for a Brain Strategy).

Also noted by a small number of respondents was the importance of the NPHSNC in highlighting a number of remaining knowledge and research gaps with respect to neurological conditions. These respondents indicated that the study will help to guide research investments of their own organization or will influence the priorities of other research granting agencies by informing grant applications related to neurological conditions.

4.5 Performance: Issue #5 - Demonstration of Economy and Efficiency

To support the economy of the NPHSNC, the program leveraged expertise and volunteer time. The study increased efficiency by connecting researchers and the neurological conditions community, thereby making study products available and relevant to a broader audience. Preliminary evidence pointed to several factors that supported the efficiency of the NPHSNC. A performance measurement culture was in place for the NPHSNC, however it was too early to demonstrate impact on decision-making.

The Treasury Board Policy on Evaluation (2009) and the guidance document, Assessing Program Resource Utilization When Evaluating Federal Programs (2013), defines the demonstration of economy and efficiency as an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes. This assessment is based on the assumption that departments have standardized performance measurement systems and that financial systems link information about program costs to specific inputs, activities, outputs and expected results.

The evaluation provided observations on economy and efficiency based on findings from the document review, key informant interviews and available relevant financial data. In addition, the findings below provide observations on the adequacy and use of performance measurement information to support economical and efficient program delivery and evaluation.

Observations on Economy

The $15 million NPHSNC investment was approximately split as follows: $1.6 million over five years went to NHCC to cover personnel, travel, overhead, equipment, material and dissemination activities; $7.2 million over five years went to fund the 13 research projects and $4.9 million over five years went to the Public Health Agency of Canada to cover personnel, evaluation, the development and execution of the surveys, progress meetings and the microsimulation project.

Planned spending and actual expenditures are presented below. Year one and year two variance can be attributed to start-up and need for a second call for proposals, and there are no salary dollars for 2013-2014 as it was expected that the study would be completed in 2012-2013.

Observations on Efficiency

Several factors helped support efficient implementation of NPHSNC objectives. First, documentary sources and the views of key informants pointed to the importance of grouping the 18 neurological conditions in a single study (compared to separate data collection for each individual condition). Many informants noted the importance of bringing together a number of diverse researchers and neurological charities as key to the success of the study.

Second, the partnership with NHCC reportedly had positive impacts in streamlining consultations and leveraging member organizations' contributions. Finally, some studies effectively demonstrated how secondary data could be used to answer research questions. Secondary data (such as systematic reviews and administrative data analyses) were less costly than primary data collection and the studies identified which secondary data sources provided the best evidence and demonstrated how other organizations could apply the methods developed in the study in other jurisdictions/neurological conditions.

Several internal program representatives mentioned that the internal monitoring of expenditures and of the progress of the component studies was very rigorous, involved the Implementation Committee and the Scientific Advisory Committee at many points during implementation, and had clear deliverables and reporting templates. The committees, where relevant, were involved in decision-making (e.g., experts reviewing proposals). A few key informants specifically mentioned the dedication and hard work of the project leads, and in particular the chairperson of the Scientific Advisory Committee (who volunteered full-time on the study). The Interim Performance Study of the NPHSNC noted several factors supporting economy, including: leveraging of volunteer time (e.g., some members of the governance committees and working groups provided their expertise and time as volunteers); and the commitment of time by researchers often exceeded the requirements in contribution agreements.

On the other hand, some internal program representative key informants reported that the partnership and coordination of the committees were labour intensive as was the creation of the Synthesis Report which involved two individuals from the Agency, members from the Synthesis Panel, the Stakeholder Engagement Panel, and the Implementation Committee, as well as external consultants.

The main challenges with the NPHSNC were related to delays including: 1) timelines for the NPHSNC that were already ambitious were exacerbated by delays in the initiation of some studies (e.g., delayed release of funds/need for second call for proposals, requirement for Research Ethics Board reviews) as well as factors outside the control of researchers (e.g., Ministry of Health activities that were suspended in one province while an investigation into a data breach was conducted); 2) challenges in the ability of a small number of component studies to meet objectives (e.g., risk factors did not yield expected evidence); and 3) delayed project deliverables that resulted in an inability for sufficient review and validation of findings (e.g., microsimulation project) and 4) Public Health Agency management turnover slowed relationship building and contingency planning.

Observations on the Adequacy and Use of Performance Measurement Data

An (evergreen) Performance Measurement Strategy was in place for the NPHSNC which was updated in 2014. The Performance Measurement Strategy included an approach to measure key program activities across the four study focus areas, as well as intended outcomes through review of study documents, progress and final reports. Contribution agreements signed with funded study proponents included a Reporting Plan which required: attendance and presentation at progress meetings, regular Progress Reports, Final Project Evaluation Report and financial information. The progress of individual studies was reviewed annually by the Public Health Agency of Canada and the Scientific Advisory Committee to discuss challenges and propose solutions. An Interim Performance Study of the NPHSNC was commissioned in 2013 which included a survey of study stakeholders, a document review and in-depth interviews. (see Appendix 5 for more details on the Interim Study)

In general, internal key informants were satisfied with the preparatory work on performance indicators and interim products such as the Interim Performance Study. The stakeholder engagement survey, as well as feedback surveys following study progress meetings provided validation for the study direction and approach. However, interviewees noted that it was still too early to demonstrate the impacts of the study on policy and practice given the recent release of the Synthesis Report and ongoing work on knowledge translation and exchange.

5.0 Conclusions

5.1 Relevance Conclusions

5.1.1 Continued Need

In 2006, the WHO estimated that neurological conditions were affecting up to one billion people worldwide and were the cause of death of an estimated 6.8 million people each year. These numbers were projected to increase as the population aged. A 2007 Canadian Institute for Health Information (CIHI) report echoed the conclusions of the WHO report, calling for collective action on neurological conditions. In Canada, little research was available at that time about the incidence, prevalence and impact of neurological conditions. Scoping of the NPHSNC design was based on extensive stakeholder input to identify research needs. While the study addressed pre-identified gaps, it also identified new knowledge gaps (e.g., the lack of data on specific populations such as First Nations, Inuit and Métis communities, children, and those with less prevalent neurological conditions).

5.1.2 Alignment with Government Priorities

Generating knowledge and understanding the burden and impact of neurological conditions was, and continues to be, a priority for the Government of Canada and the Public Health Agency of Canada (e.g., the Public Health Agency of Canada, Strategic Horizons 2013-2018, Health Promotion and Chronic Disease Prevention Branch and Centre for Chronic Disease Prevention strategic plans and operational plans). The NPHSNC was thus aligned with federal priorities and was consistent with several recently announced initiatives in the federal priority area of neurodegenerative conditions.

5.1.3 Alignment with Federal Roles and Responsibilities

The Agency's role in health promotion and protection, surveillance and disease prevention is outlined in the Department of Health Act (2006) and the Public Health Agency Act (2006). The NPHSNC is aligned with the federal government role to conduct surveillance and research, provide leadership and support knowledge translation and the role of the Public Health Agency of Canada to prevent and reduce risk for chronic diseases and injuries and (broadly) to strengthen intergovernmental collaboration on public health and facilitate national approaches to public health policy and planning. Without federal funding and leadership on the NPHSNC, there would continue to be gaps in knowledge and surveillance activities, a lack of standardized national estimates/projections, and continued limited visibility of the impact of neurological conditions. Duplication was addressed through inclusion of the key players in the federal health portfolio and the major Canadian charities in the governance of the NPHSNC (e.g., the Public Health Agency of Canada, Health Canada and CIHR and NHCC were members of the study Implementation Committee).

5.2 Performance Conclusions

5.2.1 Achievement of Expected Outcomes (Effectiveness)

The intended outcomes of the NPHSNC related to stakeholder engagement were achieved. A unique governance model that featured co-leadership of the study by the Public Health Agency of Canada and NHCC was widely praised and a broad cross-section of stakeholders was engaged at various junctures through the planning, implementation and reporting phases of the study. This approach enhanced existing partnerships or led to the development of new ones between key stakeholders, resulting in increased knowledge development and exchange.

The key study product - the Synthesis Report Mapping Connections: An Understanding of Neurological Conditions in Canada - was produced, albeit its release and complementary knowledge translation and exchange activities were delayed by several months. The study results, while only recently in the public domain, provide an enhanced and credible understanding of the incidence, prevalence, and impact of neurological conditions. The findings had value for study stakeholders who will use these data to make improvements to the care and treatment of patients with neurological conditions and will help to define future research priorities based on identified knowledge gaps. An important legacy of the study was that it enhanced the Public Health Agency of Canada's surveillance capacity to monitor four neurological conditions; including Alzheimer's disease and other dementias, Epilepsy, Parkinsonism and Multiple sclerosis.

Evidence of the use of the study results to inform program, services and policy was limited as the findings have recently been published. Efforts are now underway to promote the results to those who develop programs and policies for health services for those with neurological conditions.

These findings indicated that the NPHSNC achieved its intended immediate outcomes and the study has demonstrated the benefits from a collaborative governance model that may have applicability to other initiatives within the Agency. Further, the NPHSNC has developed planned knowledge products and there were early indications that work was being done towards achievement of its intermediate outcomes.

5.2.2 Demonstration of Economy and Efficiency

Study expenditures were generally aligned with allocated funds. While the governance model and broad engagement of stakeholders during the implementation and synthesis phases were labour intensive with associated costs, the evaluation also noted that the NPHSNC leveraged expertise and volunteer time from experts, researchers and health charities. Other efficiencies were gained through grouping the 18 neurological conditions in a single study and the extensive use of secondary data sources. Where challenges were noted, these often had to do with delays which impacted the implementation of the component studies and a small number of studies which did not achieve the intended objectives. Given the recent release of the Synthesis Report, CCDP will continue to measure performance from the NPHSNC such as enhancements to the PHAC surveillance system as appropriate.

6.0 Lessons Learned

Formal recommendations are not being proposed given that funding for the study ended in March 2014. Based on the findings and conclusions outlined in this evaluation report, here are lessons learned that may be valuable to consider in the future for similar program initiatives.

1. The NPHSNC governance model was a success story that could be applied to other Public Health Agency of Canada research initiatives. This approach enhanced existing partnerships or led to the development of new ones between key stakeholders, resulting in increased knowledge development and exchange.

   The governance model for the NPHSNC featured Public Health Agency of Canada and NHCC co-leadership of the study. The model received broad approval from both internal and external study stakeholders as an effective way to leverage the expertise of the multiple stakeholders with benefits for the overall quality of the study. While there are costs associated with the model and there were some initial challenges as study partners came to understand one another's culture, the advantages were perceived to outweigh the disadvantages. NHCC, as an umbrella organization representing the range of large and small neurological conditions charities, was an important single window. Key to the success of the governance model was early, frequent and open communications, clear roles, responsibilities and expectations, resolving issues as they arose, documenting decisions and commitment to a common goal.

2. The Public Health Agency of Canada developed the knowledge translation plan in partnership with NHCC and co-authored the report and a number of knowledge products. However, including knowledge translation in the original contribution agreement could have enhanced this process and improved implementation of this phase.

   The Synthesis Report and other knowledge translation and exchange products are being created and disseminated to support the broad distribution, increased impact and uptake of the results of the NPHSNC. All study partners are sharing in this activity. However, the knowledge translation and exchange phase of the study, specifically the participation of NHCC, was not anticipated in the original contribution agreement for the management of the study. While an amended agreement was eventually negotiated with NHCC, including it as part of the original contribution agreement would have allowed for more effective preparation for the knowledge translation and exchange phase.

3. Research timelines should adequately anticipate (or include flexibility to deal with) start-up delays related to government processes and ethical approvals in order to ensure research initiatives are able to meet objectives.

   The most frequently mentioned challenge for the study was delays that occurred in the release of NPHSNC funds and due to ethical approvals required for projects involving primary data collection. The schedule for the study research components was consequently compressed which created frustration and had a negative impact on the ability of the research projects to fully meet objectives or gave rise to missed opportunities in maximizing the use of the data collected.

Appendix 1 - NPHSNC Component Studies

(from: Mapping Connections: An Understanding of Neurological Conditions in Canada, 2014)

1. Kim Reimer
   Title: Neurological Conditions in British Columbia

   Project objectives:

   1. To provide population-level estimates of incidence, prevalence, comorbidity, mortality, health care service utilization, and health care costs for selected neurological conditions in British Columbia;
   2. To perform an extensive review of the diagnostic codes used to identify neurological conditions over time, through the consideration of alternative case definitions and through the comparison of findings with other sources, including surveys and published data.

2. Catherine Pelletier, Asako Bienek, Dr. Sulan Dai, Jay Onysko, and Chris Waters
   Title: Expansion of the Canadian Chronic Disease Surveillance System for National Surveillance of Neurological Conditions

   Project objectives:

   1. Among the 15 neurological conditions assessed, to determine which conditions can be appropriately tracked at the national level using health administrative databases;
   2. To develop case definitions for Alzheimer's disease and other dementias, epilepsy, multiple sclerosis, parkinsonism, and stroke to track national prevalence and incidence, and eventually all-cause mortality, comorbidities, and use of health care services among Canadians living with these conditions.

3. Claudia Lagacé, Asako Bienek, Catherine Pelletier, Ming-Dong Wang, Dr. Sulan Dai, Dr. Christina Bancej, Dr. Catherine Dickson, and Jay Onysko
   Title: Findings from the Canadian Community Health Survey (2010-2011)

   Project objectives:

   1. To provide community-based prevalence estimates for neurological conditions by age and sex;
   2. To examine the prevalence of neurological conditions by selected sociodemographic variables.

4. Dr. Michael Shevell and Dr. Maryam Oskoui
   Title: Expansion of a Canadian Multi-Regional Population-Based Cerebral Palsy Registry
   Short title: Cerebral Palsy Registry Project

   Project objectives:

   1. To extend the existing cerebral palsy registry to additional regions and provinces, specifically Southern Alberta, British Columbia, Nova Scotia, and Newfoundland and Labrador;
   2. To identify risk factors associated with cerebral palsy;
   3. To acquire additional information on health service delivery to young children with cerebral palsy in order to identify any potential service gaps and regional disparities.

5. Dr. Christina Wolfson and Dr. Parminder Raina
   Title: Canadian Longitudinal Study on Aging: Neurological Conditions Initiative (CLSA-NCI)

   Project objectives:

   1. To design a comprehensive assessment of four neurological conditions (Alzheimer's disease and other dementias, brain injury, epilepsy, Parkinson's disease) within the CLSA to estimate the prevalence and incidence of these conditions;
   2. To assess the impact of the four neurological conditions on health, the social care system and families using both individual level data and linkage with administrative databases;
   3. To develop CLSA sub-studies to examine the risk and prognostic factors of the targeted neurological conditions and conduct a subset of these studies within the timeframe of the Study.

6. Dr. Neil Drummond and Dr. Richard Birtwhistle
   Title: Canadian Primary Care Sentinel Surveillance Network - Neurodegenerative Conditions

   Project objectives:

   1. To develop an efficient, effective, dynamic, valid, longitudinal chronic disease database that is relevant at local, regional, provincial and national levels, with particular reference to three neurological conditions (Alzheimer's disease and other dementias, epilepsy, parkinsonism);
   2. To study the epidemiology of Alzheimer's disease and other dementias, epilepsy and parkinsonism in Canada;
   3. To study the clinical care of people with Alzheimer's disease and other dementias, epilepsy, and parkinsonism in primary care settings.

7. Dr. Susan Jaglal
   Title: Use and Gaps in Health and Community-Based Services for Neurological Populations: A Systems Analysis

   Project objectives:

   1. To identify health and community-based service needs and gaps and exemplary programs in the peer-reviewed and grey literature for individuals living with neurological conditions;
   2. To explore from the perspective of service providers, the health and community based service needs and gaps in the Canadian context taking into account different geographic regions (Atlantic Canada, Quebec, Ontario, Western Canada, Northern Canada), urban and rural, and community and institutional settings;
   3. To describe the availability, access, use, coordination and integration of health and community services and perceived needs and gaps among service providers across the continuum of care and across the lifespan, by the variables listed above;
   4. To identify opportunities and successes that can be leveraged across the regions and nationally to formulate key recommendations to help governments and stakeholders better plan programs and health services.

8. Dr. John Hirdes, Dr. Colleen Maxwell, and Dr. Nathalie Jetté
   Title: Innovations in Data, Evidence and Applications for Persons with Neurological Conditions (ideas PNC)

   Project objectives:

   1. To estimate the prevalence of 10 neurological conditions across the continuum of care in at least four Canadian provinces;
   2. To examine the costs of care associated with neurological conditions including both formal and informal sources;
   3. To examine the experience of informal caregivers providing support, including the extent and type of care provided, to different populations with neurological conditions in different care settings;
   4. To develop a detailed clinical profile of persons with at least one of the 10 targeted neurological conditions in different care settings;
   5. To evaluate the applicability of current planning approaches used in various service settings to persons with neurological conditions and provide recommendations on how to improve care planning protocols to meet current best practices identified in the neurological literature;
   6. To examine access to, and utilization of, health and social services by persons with neurological conditions to identify service gaps where the needs of this population may not be met;
   7. To identify approaches for performance measurement for organizations serving persons with neurological conditions.

9. Dr. Joan Versnel and Dr. Tanya Packer
   Title: The Everyday Experience of Living with and Managing a Neurological Condition (LINC)

   Project objectives:

   1. To describe the impact of neurological conditions on the everyday life experiences of individuals, families, caregivers and communities;
   2. To examine the complex inter-dependence between adults with a neurological condition and their families;
   3. To describe the ability of health, social and community services and agencies in supporting individuals and families to self-manage life with a neurological condition.

10. Dr. Christina Bancej, Dr. Rochelle Garner, Dr. Philippe Fines, Dr. Douglas Manuel, Anna J. Zycki, Dr. Ronald Wall, Trang Nguyen, and Julie Bernier
    Title: National Population Health Study of Neurological Conditions Microsimulation Component: Health and Economic Modelling of Neurological Conditions

    Project objectives:

    1. To build microsimulation models of seven groups of neurological conditions and injuries (Alzheimer's disease and other dementias, cerebral palsy, epilepsy, multiple sclerosis, Parkinson's disease/parkinsonism, hospitalized traumatic brain injury, hospitalized traumatic spinal cord injury), expanding on the Statistics Canada Population Health Model platform and to apply these models to project the future burden, both in terms of dollars and impact on health, of key neurological conditions over the next five, 10, 15, and 20 years under status quo assumptions.

11. Dr. Carrie Bourassa
    Title: Understanding from Within: Developing Community-Driven and Culturally Relevant Models for Understanding and Responding to Neurological Conditions among Aboriginal People (Native Women's Association of Canada)

    Project objectives:

    1. To improve the understanding of how Aboriginal Canadians conceptualize neurological conditions and the impact on their families and communities;
    2. To determine risk and protective factors for neurological conditions among the participants of this study;
    3. To assess the needed resources and supports to provide culturally appropriate care.

12. Dr. Karen Tu, Dr. Liisa Jaakkimainen, and Dr. Debra Butt Title: Validation of Administrative Data Algorithms to Determine Population Prevalence and Incidence of Alzheimer's Disease and Other Dementias, Multiple Sclerosis, Epilepsy and Parkinson's Disease (Parkinsonism)

    Project objectives:

    1. To perform an administrative data validation using coding algorithms based on case ascertainment for epilepsy, multiple sclerosis, Alzheimer's disease and other dementias and Parkinson's disease. Validated administrative data algorithms for each of these conditions will provide new opportunities to examine their prevalence and incidence among patients.

13. Dr. Daniel Krewski
    Title: Systematic Review of Factors Influencing the Onset of Neurological Conditions

    Project objectives:

    1. To conduct a comprehensive, systematic literature review for the development of each of the priority neurological conditions with respect to a wide range of risk factors, including biological, lifestyle, socioeconomic, environmental, and psychosocial factors, and comorbidity (such as stroke or mental illness) and possible mechanisms of action;
    2. To assess and summarize the available evidence on the determinants of neurological conditions, and describe the strengths and weaknesses of the current scientific literature.

14. Dr. Daniel Krewski
    Title: Systematic Review of Factors Influencing the Progression of Neurological Conditions

    Project objectives:

    1. To conduct a comprehensive, systematic literature review for the progression of each of the priority neurological conditions with respect to a wide range of risk factors, including biological, lifestyle, socioeconomic, environmental, and psychosocial factors, and comorbidity (such as stroke or mental illness) and possible mechanisms of action;
    2. To assess and summarize the available evidence on the determinants of neurological conditions, and describe the strengths and weaknesses of the current scientific literature.

15. Dr. Lawrence Korngut, Dr. Nathalie Jetté, and Dr. Tamara Pringsheim
    Title: Neurological Registry Best Practice Guidelines and Implementation Toolkit

    Project objectives:

    1. To create comprehensive guidelines through consensus building for the development and implementation of registries of neurological conditions in Canada;
    2. To create a toolkit for the development and implementation of neurological condition registries in Canada.

16. Claudia Lagacé, Asako Bienek, Catherine Pelletier, Ming-Dong Wang, Dr. Sulan Dai, Dr. Christina Bancej, Dr. Catherine Dickson, and Jay Onysko
    Title: Survey on Living with Neurological Conditions in Canada (2011-2012)

    Project objectives:

    1. To describe the characteristics of individuals with a neurological condition;
    2. To describe the general physical and mental well-being of Canadians with a neurological condition;
    3. To describe comorbid chronic conditions among Canadians living with a neurological condition;
    4. To describe the impact of neurological conditions on quality of life.

17. Claudia Lagacé, Asako Bienek, Catherine Pelletier, Ming-Dong Wang, Dr. Sulan Dai, Dr. Christina Bancej, Dr. Catherine Dickson, and Jay Onysko
    Title: Survey of Neurological Conditions in Institutions in Canada (2011-2012)

    Project objectives:

    1. To provide prevalence estimates of neurological conditions in long-term care facilities.

18. Dr. Nathalie Jetté and Dr. Tamara Pringsheim
    Title: Understanding the Epidemiology of Neurological Conditions and Building the Methodological Foundation for Surveillance

    Project objectives:

    1. To perform systematic reviews of the incidence and prevalence of all priority neurological conditions identified by the Agency;
    2. To summarize and make recommendations on the best ascertainment sources for surveillance for each of the neurological conditions of interest;
    3. To develop an inventory of existing neurological registries in Canada and other developed countries.

Appendix 2 - Description of the Logic Model for the National Population Health Study on Neurological Conditions^{Footnote v}

The National Population Health Study on Neurological Conditions (NPHSNC) uses the following Component Initiatives to deliver its activities, produce outputs and accomplish its outcomes: coordination, risk factors, scope, impact and health services.

The NPHSNC consists of eleven main activities delivered by the study and its partners, namely:

• Coordinate activities through a Project Coordination Committee, review plans, and monitor progress, KTE
• Identify risk factors for developing neuro conditions and poor outcomes
• Conduct research projects related to incidence, prevalence and co-morbidities
• Expand Canadian Chronic Disease Surveillance System to include neuro (CCDSS)
• Conduct survey of people living in long-term care institutions
• Expand Canadian Community Health Survey (CCHS) screener questions
• Conduct in-depth survey of people living with neurological conditions (SLNCC)
• Use micro-simulation models to project impacts of neurological conditions (5-20 years)
• Develop an Electronic Registry to document rare diseases, functionality and quality of life
• Identify challenges and coping mechanisms and Conduct case studies
• Identify gaps in health services and Conduct health service review

These activities are targeted at different groups, namely:

• P/T Governments, Local/Regional Public Health and Health Planning Bodies, NGOs, Academia, Clinicians, CIHR and other Research Bodies, and individuals and families with neurological diseases

As a result of each activity, the Study generates a number of products, namely:

• Terms of reference
• Work plan
• Minutes
• Report on risk factors
• Final project reports
• Report on incidence of neurological diseases
• Report on the characteristics of persons living in long-term care institutions
• Report on prevalence of neurological diseases
• Report on individual and family impacts of living with neurological diseases
• Projection report on the economic impact and need for health services over time
• Electronic Registry Prototype
• Information products for people living with neurological diseases (i.e., coping strategies)
• Report on health services, quality of life and functional assessment tools for the clinical setting
• Study information dissemination
• NHCC and PHAC website, PHAC Comprehensive Surveillance Report, New research questions, Peer review journal articles, Synthesis Meeting, Consensus Conference, NPHSNC Report to the Minister

These activities correspond with specific immediate, intermediate and longer-term outcomes. In the immediate term, program activities are expected to lead to engagement of stakeholders. The achievement of this immediate outcome is expected to lead to the intermediate outcome of use of NPHSNC knowledge products. In the longer-term the NPHSNC expects to have the capacity to plan programs, services and policies to prevent or manage neurological diseases.

The ultimate outcome of the Study is to reduce the impact of neurological disease in Canada.

Appendix 3 - Summary of Findings

Rating of Findings 

Ratings have been provided to indicate the degree to which each evaluation issue and question have been addressed.

Relevance Rating Symbols and Significance: 

A summary of Relevance ratings is presented in Table 1 below. A description of the Relevance Ratings Symbols and Significance can be found in the Legend.

Performance Rating Symbols and Significance: 

A summary of Performance Ratings is presented in Table 2 below. A description of the Performance Ratings Symbols and Significance can be found in the Legend.

Appendix 4 - Evaluation Description

Evaluation Scope

The scope of the evaluation included an assessment of the relevance and performance of the NPHSNC from program inception, 2009-2010 to 2013-2014.

Evaluation Issues

The specific evaluation questions used in this evaluation were based on the five core issues prescribed in the Treasury Board of Canada's Policy on Evaluation (2009). These are noted in the table below. Corresponding to each of the core issues, evaluation questions were tailored to the program and guided the evaluation process.

Data Collection and Analysis Methods

Evaluators collected and analyzed data from multiple sources. Sources of information used in this evaluation included the following.

Document and File Review

All documents provided by the Project Authority for the purposes of the evaluation were reviewed to provide a foundation for the evaluation and contribute a line of evidence that addresses most evaluation questions. The following types of documents were reviewed: program- and project (study)-level descriptive and administrative materials; Government and departmental level policy and planning documents; and evaluation and performance reports: to maximize the efficiency and utility of the document review, a document review template was developed to facilitate the systematic review of materials.

Key Informant Interviews

The main purpose of the interviews was to gather information to fill gaps identified in the document/file review and to provide evidence and detailed information to help contextualize evidence gathered from other sources. Tailored guides were developed to be suitable for administration with two groups of key informants: internal respondents from the Public Health Agency who were involved in the design and delivery of the NPHSNC; and external stakeholders, specifically representatives of Neurological Health Charities Canada, members of the various governance committees (i.e. the Scientific Advisory Committee and Synthesis Panel, Stakeholder Engagement Working Group), and project leads who received funding from the NPHSNC. In total, five interviews with seven internal program representatives were conducted and 15 interviews were conducted with external stakeholders.

Data were analyzed by triangulating information gathered from the different sources and methods listed above. This included: systematic compilation, review and summarization of documentary sources, quantitative analysis of analytics data; thematic analysis of qualitative data from key informant interviews; and comparative analysis of data from disparate sources to validate summary findings.

Appendix 5 - Interim Performance Study

EXECUTIVE SUMMARY

INTRODUCTION

The National Population Health Study of Neurological Conditions (NPHSNC) is a five-year, $15 million study slated for completion in March 2014. The purpose of the NPHSNC is to provide key data and information on neurological conditions in Canada. The Government of Canada (GoC) provided funding for the NPHSNC, and an Implementation Committee (IC) oversees the project (with representatives from the GoC health portfolio - Public Health Agency of Canada, Health Canada, and the Canadian Institutes of Health Research - and the Neurological Health Charities Canada).

The NPHSNC funded 13 research projects through contribution agreements, with additional microsimulation work led by PHAC in collaboration with Statistics Canada. Three surveys were also developed as part of the NPHSNC. The Scientific Advisory Committee (SAC) provided scientific advice and oversight on the various projects.

OBJECTIVES OF THE PERFORMANCE STUDY

The purpose of this interim performance study was to assess the following three areas at this stage of the NPHSNC:

1. lessons learned to date;
2. the effectiveness of the unique governance approach (NGOs and government working together) and implementation process; and
3. stakeholder engagement (to date) over the course of the initiative.

METHODS

• The interim performance study took place between December 2012 and May 2013 while the NPHSNC was still underway.
• Multiple methods with multiple stakeholder groups were employed.
• Sampling strategies were either a census of all participants or ensured maximum variation to minimize any potential bias.
• The evaluation was conducted by an independent consultant.
• Background interviews with 8 past and present members of the IC/key project staff, as well as document review, were conducted to gather initial data and inform the performance study plan, sampling decisions and tool development.
• In Stage One, an on-line survey on partnership engagement ("Engagement Tool"), developed by the Centre for Chronic Disease Prevention's (CCDP) Program Performance Section, was used to survey past and present IC members and key staff (with 18 out of 24 responses). Initial qualitative telephone interviews were conducted with 3 Principal Investigators (PIs) and 3 members of the SAC.
• In Stage Two, following the administration of the Engagement Tool, 23 past and present IC members and key staff were interviewed by telephone. In addition, an on-line survey was developed for SAC members and PIs/research teams, with 7 SAC members (out of 9) and 16 PIs/research teams completing the survey.
• While there was a high response rate for each method employed, a few people chose not to participate in the interviews or surveys.

FINDINGS

Lessons Learned from Study to Date

• Across the three groups (IC, SAC, and PIs/research teams), 76% rated the overall success of the NPHSNC as excellent (15/41) or good (16/41). The remaining 9/41 (22%) rated it fair and 1 person (2%) rated it as poor.
• While, overall, those participating in the interim performance study felt that the NPHSNC was a success, some cautioned that it was early to assess success given the research projects had not been submitted and synthesis work still remained.
• It was felt the NPHSNC resulted in a higher return on investment than the $15 million in contributions, due to volunteer time as well as researcher involvement above and beyond their funding requirements.
• Most felt that critical evidence gaps would be filled by the work of the NPHSNC; however, the nature of research is such that further critical gaps were identified as part of the work. The NPHSNC did not cover all neurological diseases and a few interviewees noted gaps remained for certain population groups (e.g. Aboriginal populations).

Unique Governance Approach

• The collaborative approach was identified as being a key to the success of the NPHSNC.
• Champions from within government and the neurological health charities had the vision and leadership to operationalize the NPHSNC.
• The federal government and the neurological health charities each brought different strengths, expertise and perspectives. The charities had a consumer perspective and connections to researchers, while the government brought funding, policy perspective and existing systems. It was felt this comprehensive approach may facilitate knowledge translation of the findings and increase impact of the work.
• While there were some cultural differences between the charities and the government, stakeholders were highly committed to resolving issues as they arose.

Stakeholder Engagement To Date

• There was general agreement that there was sufficient common ground amongst stakeholders to work together.
• Despite a few communication and collaboration challenges, across the three groups (IC, SAC, and PIs/research teams), 85% strongly agreed (12/41) or agreed (23/41) that those working collaboratively on the NPHSNC are able to achieve things they could not achieve as readily working alone.

LESSONS LEARNED

• It is clear from the interim performance study that, to date, there has been much success in implementing the NPHSNC.
• This complex study has largely been implemented as planned (including funding key research projects, engaging multiple stakeholders, and augmenting the profile of neurological diseases in Canada), and has resulted in successful collaborations amongst various researchers, the federal government and neurological health charities.
• It is felt that the resulting research studies will be of key importance to addressing the challenges of neurological diseases in Canada.
• While some challenges were identified in the governance approach of NGOs and government working together, these were largely addressed throughout the course of the NPHSNC's implementation. These learnings (including ensuring ongoing open communication, clearly documenting decisions, and ensuring clear roles and responsibilities) may be useful for any future work using a similar template.
• A number of administrative challenges were identified based on PHAC's funding of research projects, including multiple Research Ethics Board reviews, delayed research funding with no timeline extensions, and the administrative burden of the contribution agreement mechanism. These challenges should be considered in any future funding of research projects that PHAC undertakes.

POTENTIAL NEXT STEPS

The current evaluation was an interim performance study. It is suggested that the next phase of the evaluation focus on outcomes following the completion of the NPHSNC in March 2014. This next phase could further examine engagement, as well as evaluate outcomes from the study related to use and uptake of findings.

Appendix 6 - References

• Alzheimer Society Canada. (2014). About Dementia Friends. Retrieved from http://www.alzheimer.ca/en/ News-and-Events/feature-stories/dementia-friends
• Canadian Institute for Health Information. (2007). The Burden of Neurological Diseases, Disorders and Injuries in Canada. Retrieved from https://secure.cihi.ca/free_products/BND_e.pdf

• Government of Canada. (2014). Canada and France Encourage Global Action on Dementia. Retrieved from http://news.gc.ca/web/article-en.do?mthd=advSrch&crtr.mnthndVl=9&crtr.mnthStrtVl=1&crtr.page=1&nid=883589&crtr.yrndVl
  =2014&crtr.kw=Dementia&crtr.yrStrtVl=2002&crtr.dyStrtVl=1&crtr.dyndVl=12

• Government of Canada & National Health Charities Canada. (2014). Mapping Connections: An Understanding of Neurological Conditions in Canada. Retrieved from http://www.phac-aspc.gc.ca/publicat/cd-mc/mc-ec/assets/pdf/mc-ec-eng.pdf
• Grant, J., Cottrell, R., Cluzeau, F., & Fawcett, G. (2000). Evaluating "payback" on biomedical research from papers cited in clinical guidelines: applied biometrical study. British Medical Journal, 320, 1107-1111.
• Morris, Z.S., Wooding, S., & Grant, J. (2011). The answer is 17 years, what is the question: understanding time lags in translational research. Journal of the Royal Society of Medicine, 104, 510-520.
• Neurological Health Charities Canada. (2014). Background. Retrieved from http://www.mybrainmatters.ca/en/background
• Neurological Health Charities Canada. (2014). Landmark Study Reveals Impacts of Neurological Conditions on Canadians. Retrieved from http://www.mybrainmatters.ca/en
• Neurological Health Charities Canada. (2014). National Population Health Study of Neurological Conditions. Retrieved from http://www.mybrainmatters.ca/en/national-population-health-study-neurological-conditions
• Neurological Health Charities Canada. (2013). National Population Health Study of Neurological Conditions: Frequently Asked Questions. Retrieved from http://www.mybrainmatters.ca/sites/default/files/FAQ%20for%20NPHSNC-June2013.pdf
• Public Health Agency of Canada. (2014). Fact Sheet - Mapping Connections: An Understanding of Neurological Conditions in Canada. Retrieved from http://www.phac-aspc.gc.ca/cd-mc/nc-mn/mc-fs-ec-fr-eng.php
• Public Health Agency of Canada. (2014). Multi-sectoral Partnerships to Promote Healthy Living and Prevent Chronic Disease. Retrieved from http://www.phac-aspc.gc.ca/fo-fc/mspphl-pppmvs_s10-eng.php#q1
• Public Health Agency of Canada. (2014). Neurological Conditions. Retrieved from http://www.phac-aspc.gc.ca/cd-mc/nc-mn/index-eng.php
• Public Health Agency of Canada. (2014). Strategic Horizons 2013-2018. Retrieved from http://publications.gc.ca/collections/collection_2014/aspc-phac/HP55-1-2013-eng.pdf
• Public Health Agency of Canada. (2014). The National Population Health Study of Neurological Conditions - Governance of the study. Retrieved from http://www.phac-aspc.gc.ca/cd-mc/nc-mn/gov-dir-eng.php
• Public Health Agency of Canada. (2014). The National Population Health Study of Neurological Conditions - Main Approaches. Retrieved from http://www.phac-aspc.gc.ca/cd-mc/nc-mn/ma-pa-eng.php
• World Health Organization. (2006). Neurological Disorders: public health challenges. Retrieved from http://www.who.int/mental_health/neurology/neurological_disorders_report_web.pdf

Footnotes

List of Acronyms

ALS

Amyotrophic Lateral Sclerosis

CCDSS

Canadian Chronic Disease Surveillance System

CEEHRC

Canadian Epigenetics, Environment and Health Research Consortium

CCNA

Canadian Consortium on Neurodegeneration and Aging

CIHI

Canadian Institute for Health Information

CIHR

Canadian Institutes of Health Research

CLSA-NCI

Canadian Longitudinal Study on Aging: Neurological Conditions Initiative

Gs&Cs

Grants and Contributions

ICRSAD

International Collaborative Research Strategy for Alzheimer's Disease

INMHA

Institute of Neurosciences, Mental Health and Addiction

ISHLCD

Integrated Strategy on Healthy Living and Chronic Disease

NHCC

Neurological Health Charities Canada

NPHSNC

National Population Health Study of Neurological Conditions

O&M

Operations and Maintenance

PAA

Program Alignment Architecture

RMNI

Regenerative Medicine and Nanomedicine Initiative

WHO

World Health Organization

Endnotes